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Abstract 

We investigate the maximum number of embedded patterns in the two-dimensional Hopfield 
model. The grand state energies of two specific network states, namely, the energies of the pure- 
ferromagnetic state and the state of specific one stored pattern are calculated exactly in terms of 
the correlation function of the ferromagnetic Ising model. We also investigate the energy landscape 
around them by computer simulations. Taking into account the qualitative features of the phase 
diagrams obtained by Nishimori, Whyte and Sherrington [Phys. Rev. E 51, 3628 (1995)], we 
conclude that the network cannot retrieve more than three patterns. 
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I. INTRODUCTION 



The Hopfield model |T[ is one of the simplest mathematical models which explains as- 
sociative memory. This model is characterized by binary state neurons and each neuron is 
represented by Ising spin. In this model system, arbitrary two neurons are interacted each 
other via so called Hebb rule. The Hebb rule is one of the standard learning rules of the 
patterns (i = 1, . . . , A^; /i = 1, . . . ,p) and determines the strength of the interaction Jjj 
between the i and j-th neurons, say. Si and Sj, as 

^^. = ^EC%^ (1) 

where /i means the number of embedded patterns and denotes the number of neurons. 
These features have been investigated by statistical mechanics deeply [|], |^. Actually, up to 
now, various extensions and generalizations of the Hopfield model were proposed and these 
properties were investigated from statistical mechanical point of view (see, for example 0). 
However, little is known about the properties of the Hopfield model in which the length of 
interactions Jij is restricted to the nearest neighboring neurons. By the analogy to the spin 
system in statistical mechanics, we call this type of the Hopfield model finite- dimensional 
Hopfield model. Inspired by the study of Nishimori, Whyte and Sherrington in this 
paper, we consider the finite-dimensional Hopfield model storing structured patterns. In 
general, it is hard to analyze such finite-dimensional systems explicitly. However, one can 
derive several rigorous results of thermodynamic properties of the system by assistant of the 
Pierls arguments and the gauge transformations [Q. Although the qualitative features of 
the phase diagrams of the system became clear by these analyses, nobody yet succeeded in 
deriving their quantitative behavior at all. In this paper, we analyze the storage capacity of 
the system quantitatively. We restrict ourselves to the case of two-dimensional system on 
the square lattice. 

This paper is organized as follows. In Sec. ||, we explain our model system. In Sec. IH, 
we briefly review the qualitative features of the phase diagram obtained by Nishimori et al 
0. In Sec. |IV|, we analyze the storage capacity of our model system. Section |V| is devoted 
to discussion of all results we obtain. 
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II. DEFINITION OF THE SYSTEM 



In this section, we define our model systems. The Hamiltonian of the system is given as 

H=-^JijSiSj, (2) 

{ij) 

where Si{i = 1, . . . , A^) are the states of the neuron taking binary value ±1, and Jij is the 
strength of the interaction between Si and Sj. The summation appearing in Eq. (H) 
runs over nearest neighboring neurons on a square lattice. We chose the short range Hebb 
rule as a interaction which is given by 

J^^ = ^tm (3) 

for the nearest neighboring sites (ij) and Jij = otherwise. Here, p is the number of 
embedded patterns and = ±1 (/i = 1, . . . ,p; z = 1, . . . , A^). Let us consider about the 
probability distribution of a set of patterns {^f }(= {^f | i = 1, . . . , N; ft = 1, . . . ,p}). We 
suppose that the probability that arbitrary nearest neighboring sites of /i-th pattern are 
and respectively is proportional to 

A 

where the parameter Jq controls the degree of the correlation between arbitrary nearest 
neighboring sites. Let us write P(^f = and P{C,i = as the probability of = 
and = respectively. Then, the ratio of the former to the later is given by 

' - exp — . (5) 



exp ( ^ere; , (4) 



For the case of Jq = 0, Eq. becomes P(^f = ^j) = = Hence there is 

no correlation between i and j sites of /i-th patterns and embedded patterns correspond to 
"random patterns" . On the other hand, for the case of Jq ^ oo, we obtain = = 0. 
Namely, the value of is same as that of with probability one. Applying Eq. @ to all 
nearest neighboring pairs (ij), we obtain the probability distribution for /x-th pattern as 

^ expfA^^M^^V (6) 
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where Zq{Jq/ ^) is the normahzation factor given by 

Zo{^=Y.^^vi^Y.^^i^■ (7) 

Supposing that each pattern is generated by Eq. (|) independently, a set of embedded 
patterns {^f } is generated by the following probability distribution 

pm}) = c\{^^v{^Y.^^i^- (8) 

Here, the normalization factor c is given by {Zo( Jo/y^)}^- Note that Eq (^) corresponds 
to the Boltzmann weight of the two-dimensional ferromagnetic Ising model on the square 
lattice whose interaction is given by 1/^ at temperature 1/Jq. Hence embedded patterns 
are same as snapshots of equilibrium Monte Carlo simulations for it. This Ising model was 
explicitly solved in and it is known that there is a critical point aX K = Kc = 0.44, where 
K denotes the ratio of the temperature to the strength of the interaction. This model has a 
ferromagnetic solution for Jq/a/P > -^c and a paramagnetic solution for Jo/^/P < K^. From 
this fact, in our model system, embedded patterns have a long-range order for Jq > -^ca/P, 
on the other hand, there is no long-range correlation for Jq < K^^. Fig. shows typical 
three examples of embedded patterns. 

In the next section, we briefly review the features of the phase diagrams of our model 
systems obtained by Nishimori et a/ |Q. 

III. GENERIC QUALITATIVE PHASE DIAGRAM 

Before we explain our analysis of maximum number of embedded patterns, in this section, 
we briefly review the results by Nishimori et a/ 0]. Note that their treatments, namely, the 
gauge transformations and the Peierls arguments are applied to not only the two-dimensional 
systems but also the systems in arbitrary dimension. Fig. ^ shows the qualitative phase di- 
agram with axes of temperature T and a parameter Jq controlling the structure of patterns 
for a fixed value of p. In general, this system has three phases: paramagnetic(P), ferromag- 
netic(F), and retrieval(R)(or spin glass(SG)) phases. Each phase is characterized by the 
following three order parameters 



1 ^ 



= j^H^S,), (10) 



N 



mR = ^Eer(^.), (11) 

i=l 

where (■ ■ ■) means thermodynamic average. The above three order parameters q, mp and 
iriR represent the Edwards-Anderson spin glass order parameter, the ferromagnetic order 
parameter, and the overlap between the /i-th pattern and the network state {Si\, respec- 
tively. The paramagnetic, ferromagnetic, retrieval and spin grass phase are characterized by 
the above three order parameters as g = mp = ruR = 0; g > 0, mp > 0; g > 0, uir > and 
q > 0,mF = rriR = 0, respectively. 

Applying the gauge transformations to this system, we obtain the internal energy [{E)] 
and the overlap [m/j] on the lines (3{= 1/T) = Jq in the phase diagram as follows. 

m] = pEo , (12) 

M = ^0 (^-^ , (13) 

where i?o(/5/v^) is the internal energy of the ferromagnetic Ising model corresponding to 
the partition function Zo{f3/y/p), and fno{(]/y/p) is the spontaneous magnetization of the 
same model. Here, the expression of [■ ■ ■] means the average over the distribution (|^). Both 
Eq^P/^/p) and "^o(/5/^/p) generally have singularities at some critical point when spatial 
dimensionality exceeds one. Let us suppose that P/y/p = Kc is the critical point. Then 
Eqs (|12D and ([13|) mean that the internal energy and the overlap of our system have the 
same singularity at {3/ ^ = Kc- This implies a phase transition and the boundary between 
[itT'r] 7^ and [rriR] = crosses this point on the line /3(= 1/T) = Jq (it is denoted by M in 
Fig. I). We should notice that this point is also the critical point of the embedded patterns. 

When the value of p is small, a typical phase diagram is given by Fig. H(a). For this case, 
there is a possibility to exist the retrieval phase in the region denoted by R. In the same 
figure. The region F' is the ferromagnetic phase with finite overlap. It is important to notice 
that this system is not meaningful as an associative memory for Jq > K^.^ since embedded 
patterns have a long-range ferromagnetic order like the right of Fig. |1|, and patterns become 
correlated each other. With this fact in mind, we do not regard this region as a retrieval 
phase. When the value of p increases, the critical point Jq = -ft'ci/P nioves to right. On the 
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other hand, by using the Peierls argument, the ferromagnetic phase still exists in the almost 
same region. Taking those facts into account, there exists a critical number of patterns 
Pc above which {p > p^) the ferromagnetic phase is split into two regions, that is, the 
ferromagnetic phase with finite overlap (F') and the ferromagnetic phase without retrieval 
order (F). At the same time, the retrieval phase in the small-p case is replaced by the spin 
glass phase because the region with finite overlap is limited to F' (Fig. 0(b)). In summary, 
there exists a critical number of patterns pc blow which {p < pc) the retrieval phase exists 
there if any, while for p > p^ the retrieval phase vanishes and this network cannot retrieve 
embedded patterns. 

However, the value of Pc is not yet evaluated quantitatively at all in [Q. Following section, 
we investigate it especially in the case of two-dimensional system on the square lattice. 

IV. ANALYSIS OF THE SYSTEM 

A. The case of p = 1 

We begin with the case of j9 = 1. In this case, the system is identical to the ferromagnetic 
Ising model and the retrieval solution of the system corresponds to the ferromagnetic solution 
of the ferromagnetic Ising model. In the case of square lattice, the critical temperature of 
the ferromagnetic Ising model is = 2.27, therefore the system has a retrieval solution at 



B. The case of p > 3 

We next analyze the case of p > 3. There is a good evidence to show that the retrieval 
phase does not exist for this case. Let us start with investigating if there is a state which 
has a smaller energy than retrieval one. Substituting Si = ^li^ioi all i) into Eq. (0) and 



T<Te. 



averaging it over the distribution (H), we obtain the energy per neuron of the pure retrieval 



state 



1 



1 



p 



[Er] 



N ^ 




(14) 
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where Ci{Jq/ ^yp) is the nearest-neighbor correlation function of the ferromagnetic Ising 
model on the square lattice. The explicit form of Ci is 



(15) 



We also rewrite the energy per neuron of the pure ferromagnetic state in terms of Ci{Jo/ ^/p)■ 
Substituting Si = l(for all i) into Eq. (0) and averaging it over the distribution (H), we obtain 



1 

N 



1 ^ 



(16) 



We next investigate the properties of the function Ci{Jq/ It is written in terms of 
the partition function (|^) as follows. 



1 91ogZo(Jo/VP) 



It is important to bear in mind that log Zq{Jq/ ^) is explicitly solved in as follows. 



(17) 



where 



— log Zo l^-^j = log cosh -^j +— log{l- 4:KcosuJiUJ2)duJiduj2, (18) 

_ tanh(2 Jo/ ^) 



cosh(2Jo/v^) 

Substituting 2 cosh /i = 1/2k| coscjiI into Eq. (|18D and using 



/ log(2 cosh/i — 2 cosu;)(ict; = 27r/i, 
Jo 



/i = cosh = \og iy + Jy^ - 1 



4:K,\ COSWil 



(19) 

(20) 
(21) 



we obtain 



1 log Z„ ( A) = log (2 cosh ?|) + ^ log i (1 + v'l-(4M^-V) rf^. (22) 

Substituting Eq. ( p2D into the right hand side of Eq. (p!7D, explicit solution of Ci{Jo/y/p) is 
written by 



- coth — - 
2 VP 



1 + -k[L 

TT 



(23) 
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where L is the complete elhptic integral, namely, 



1 — Ki sin 



(24) 



with Ki = 4k and 

2Jf 



2tanh2^-l. (25) 
VP 

Fig. 1^ shows the shape of Eq. (|23|) . From Eqs. (p!4D , (p!6|) and (p3D , we obtain rigorous values 



of grand state energies of the pure retrieval and the pure ferromagnetic states. 

Now we calculate the condition that the pure ferromagnetic state has a smaller energy 
than the retrieval one, that is to say, the condition for [Er] > [Ep]. We rewrite this inequality 
by using Eqs. ([T^) and (|16]), then we obtain 

^ <cJ4^<l. (26) 



P-1 Vv^y 

As Cf ^ is a monotonically increasing function and Cf ^(1) = oo (see Fig .^), Eq. (^) leads 

to 

where Cf ^ denotes the inverse of the function Ci. We find that the pure ferromagnetic state 
has a smaller energy than the pure retrieval one as long as this condition is satisfied. As 
Jo is finite, p > 3 must be satisfied because is a monotonically increasing function and 
Cf ^(1) = oo. Further, it is easy to confirm the following inequality. 

SJlEflflM < (28) 

dp 

Taking into account [Ep] — [Ej^] < under the condition (UTf) and p > 3, [Ep] gets smaller 



relatively as p gets larger. As the result, the pure ferromagnetic state has a smaller energy 
than the pure retrieval one for T = 0, Jq > U^/p and p > 3. The value of U is 0.38 for 
p = 3, and U < 0.38 for p > 3 because of the monotonically increasing property of Cf^. 
Note that U is always less than Kc = 0.44. 

We investigate the energy landscape around pure retrieval state further. Let us calculate 
the energy increase per interaction when the state changes from pure retrieval one. The 
energy stored in an interaction is given by 

-JrASj = -^t^'^'S,S,. (29) 
VP (1=1 



Substituting Si = Sj = into Eq. p9|), we obtain the energy stored between i-th and j- 
th neurons of the network retrieving 1-th pattern completely. As the same way, substituting 
Si = Q, Sj = —^j into Eq. (|29| ) gives the energy of the interaction for the case where j-th 
neuron changes from retrieval state. Subtracting the former from the latter, and averaging 
it over the distribution (^, we obtain the energy increase per interaction when the state 
changes from retrieval one. 

The value of [dE^ is always positive, and it is expected that the energy increases on the 
average when the state changes from pure retrieval one. We estimate the energies of the 
finite overlap states as follows. Let n be the number of neurons whose states are opposite 
to the retrieval states. The relation between n and mn is given by 

^ = 1^ . (31) 

If n is not large, the number of interactions between the retrieval state and opposite state 
is expected to be about 4n because the number of nearest neighboring neurons is 4 and 
neurons whose states are opposite to retrieval states lie sparsely almost all case. Therefore, 
the energy of the system per neuron whose overlap is mR is expected to be 

An 

[E.^^] = [En] + —[dER] 

= [ER]+2{l-mn)[dEn]. (32) 

Fig. ^ shows the result of numerical simulation and compare it with Eq. (|32D. We set 
parameter values as p = 3 and Jq/^/p = 0.39(> U) at the simulation. We generate 10^ 
sample states for each value of mn and evaluate [-Emji]- Eq. (^) agrees with the result of 
the simulation near mpt = 1, although it does not agree with the simulations for the case of 
small rnpi- This is clearly because the number of interaction between the retrieval state and 
opposite state is smaller than 4n for large n. Regardless of it, the energy tends to increase 
as mji gets smaller, and the state near m^j = 1 is likely the smallest energy among the 
finite overlap states. Almost same argument can be applied to the ferromagnetic state, and 
the state near mj? = 1 is the smallest energy among the ferromagnetic order. Taking into 



account [Er] > [Ep], the energy of ferromagnetic order is expected to be smaller than that 
of retrieval order as long as p > 3 and Jq > U ^/p are satisfied. 

We also investigate the stability of pure retrieval state. The variance of cIEr is given by 



[{AidEn)}'] = [dEl] - [dEn 



2 



2 




Using Eqs. ([1^) and (p3D, The ratio of [dE^]'^ to this variance is obtained by 
[{AidEn)?] _ - 1) {l + - 2)C^i ii^y -ip- l)C. (^)'} 



1 + 2(p - i^)' + {p- lyCr 



(34) 



Partially differentiating above equation by p (the value of Ci is fixed) , we obtain the following 
inequality. 



d [[{AidEn)} 



21 



> (35) 



dp [ [dEn]^ 

This means that the more the value of p gets large, the more the energy is likely to de- 
crease when the state changes from pure retrieval one, although it increases on the average. 
Therefore, the pure retrieval state becomes unstable for large p. We can roughly estimate 
the value of p for which the pure retrieval state gets unstable by 

In Fig. ^ we plot the minimum value of p that is satisfied with Eq. (p^. For example, 
Eq. ( p^ ) is satisfied with p > 8 for Jo/Vp ~ 0.43. Note that this is probably overestimating 
and the value of p is likely smaller. We confirm this argument in numerical simulations of the 
dynamics at T = 0. Fig. ^ shows a typical example of how the overlap m^j depends on Monte 
Carlo step in computer simulations, in which = 100 x 100 and Jo/Vp = 0.43. The state 
of network starts with the embedded pattern itself. In this simulations, the pure retrieval 
state is already unstable for p = 3, and the network falls into a certain local minimum near 
rrifi = 1. This implies that the pure retrieval state is no longer stable for p > 3. 

Taking the above all results into account, we show that there is no retrieval phase if 
p > 3. As we have seen, U is always smaller than Kc = 0.44 for p > 3, and the region of 
Jq > U at T = is not retrieval phase but must be ferromagnetic one. This is because the 
ferromagnetic states have smaller energy than the retrieval one, and the state with minimum 
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energy appears with probability one for T — in terms of the Boltzmann distribution. 
Therefore, the boundary of [m/j] = is prohibited from landing at Jq < on the Jo 

axis like L2 in Fig. and it should land at Jq = ^c\fv from point M vertically like L\. 
Hence, there is no retrieval phase in the region Jq < (for all T) because the region 

with finite overlap is limited at Jq > Kc^. As a result, the phase diagram should be like 
Fig. 0(b) and we conclude that the retrieval phase does not exist for the case of p > 3. 

C. The case of p = 2 

For the case of p = 2, we cannot immediately conclude that there exists the retrieval 
phase although the energy of a perfect retrieval state is smaller than that of ferromagnetic 
one. We have to compare the energy of the retrieval state with that of the spin glass state. 
However, it is very hard to calculate the grand state energy of the spin glass state because 
the Hamiltonian of this system has a very complicated energy landscape. For this case, 
in order to evaluate the grand state of the Hamiltonian, we should carry out simulated 
annealing, for example. However, up to now, we do not yet obtain reliable results. This will 
be our future problem. 

V. CONCLUSION 

In this paper, we investigated the maximum number of embedded patterns in the two- 
dimensional Hopfield model storing structured patterns. As a result, we found that this 
system can retrieve a pattern at T < 2.27 in the case of p = 1, however the retrieval phase 
does not exist for the case of p > 3. In other words, this system cannot retrieve more than 
three patterns whose value is independent of the size of the system. This result agrees with 
a consideration of networks with randomly diluted synapses, in which is proportional to 
the average connectivity per neuron Namely, we can see from this argument that the 
value of pc for our network is finite because the average connectivity per neuron is four (finite 
number). However, our result is more strict than that from this argument. 

Moreover, ^ 3 is satisfied with any value of Jq (even the conventional "random pat- 
terns"). This implies that the smallness of the value of pc is due to the dimensionality 
two of the network rather than spatially correlated patterns. This argument also implies 
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that although there are several modifications in order to improve storage capacity with spa- 
tially correlated patterns (see |0, for example), these modifications are not expected to be 
remarkable improvements for our model system. 

From above all arguments, We conclude that storage capacity of associative memory is 
strongly restricted by the spatial structure of the network. 
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FIG. 1: Typical examples of embedded patterns. The size of patterns is 100 x 100. From the left 
to the right, the values of parameter Jo/y^ are 0.10, 0.42 and 0.60. 

FIG. 2: The qualitative phase diagram with axes of temperature T and a parameter Jq controlling 
the structure of patterns. The value of p is fixed, (a) For small p, there are three phases: para- 
magnetic P, retrieval R, and ferromagnetic with finite overlap F'. (b) For large p, there appears 
a ferromagnetic phase without retrieval order F, and the retrieval phase in the small-p case is 
replaced by the spin glass phase. 

FIG. 3: The shape of the correlation function Ci{x) . 

FIG. 4: The average energies E of the retrieval states with overlap m^. The parameter values are 
p = 3 and Joy/p = 0.39. The errorbars are results of the numerical simulation in which the size of 
network is = 100 x 100, and calculated by averaging of 10^ samples. The line shows the result 
from Eq. (H). 

FIG. 5: The minimum value of p that is satisfied with Eq. ( p^ ) as a function of the parameter 
Jq/ y/p which determines the structure of patterns. 

FIG. 6: The typical example of how the overlap m/j depends on the Monte Carlo step t in Monte 
Carlo simulations at T = 0. We set the system size and the parameter as = 100 x 100 and 
Jq/ ^/p = 0.43, respectively. 

FIG. 7: In the case of p > 3, C/ is smaller than Kc^/p and the region of Jq > t/ at T = is not 
retrieval phase. Therefore, the boundary of [m/j] = like L2 is prohibited and it should be like Li. 
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